WIP watch diff with upstream main branch - #6

…ion of the dataset)

…ce#1006)

…q len (131072) is larger than the maximum number of tokens that can be stored in KV cache (130944). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine"

… caching

…and new version of the dataset is different)

…t it has eos_token_id)

…o avoid failures or NaN

- score the query as the continuation when choices is None (the_pile/wikitext)
- shift logits by one so tokens line up with their predictions
- stop the sample cache dropping PERPLEXITY results
- add a rolling-perplexity test and check the cached wrapper returns all docs

litellm.completion expects an int, not a (N,) tuple.

Current RAG-style tasks need the row-specific retrieved context to
live in the system role, not prepended to the user query. Opt-in
flag keeps all existing tasks unchanged.

…ge LLM (to avoid some memory errors)

…ntually called)

squad_v2 was filtering out questions with no answer, which is
exactly the half of the dataset that tests refusal behavior.
Replace the filter with an explicit "unanswerable" choice.

…options, not all the possible ones. Also increase generation_size from 100 to 1024 (for thinking models)

The generator had been narrowed to MCFFormulation + the ALL label only,
which dropped the _cf/_hybrid variants and the CA/CS/UNK labels. Restore
the full formulation list and sensitivity labels.

…dict=False to get token ids, not a BatchEncoding)

…uggingface#1264)

…huggingface#1067 regression)

…text' setting

…ggingface#1271)

Under an accelerate data-parallel launch every rank holds the full gathered
results and wrote the same parquet cache file concurrently, corrupting it and
making subsequent loads fail. Write the cache only on the main process and add
a barrier so the other ranks wait for that write before reading. Add a
regression test.

Fixes huggingface#1102

…match of the instruction)

…#1326)

* Store provider credentials as SecretStr in model configs

LiteLLMModelConfig.api_key and TGIModelConfig.inference_server_auth were
plain str fields, which meant they were retained in plaintext wherever a
model config gets serialized (e.g. EvaluationTracker.results). Switch
both to pydantic SecretStr, which masks the value in reprs and default
serialization, and additionally exclude them explicitly when building
the results dict as a second layer.

The real value is still unwrapped via get_secret_value() at the specific
call sites that need it for the actual outgoing request. Added
regression tests asserting the credential never appears in the
serialized results dict.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Store JudgeLM credentials as SecretStr

JudgeLM.api_key was a plain str consumed directly by several backend
clients (OpenAI, AsyncInferenceClient, litellm). Wrap it in SecretStr on
assignment and unwrap via get_secret_value() at each usage site, for
consistency with the other model configs and to remove any reliance on
incidental string formatting to keep it out of logs or serialized
output.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Fix pre-existing ruff format drift in README and docs

Unrelated cleanup so CI's Quality check is green on this branch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Nathan Habib <nathan_habib@Mac.lan>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Bring in 19 upstream commits (up to 932e1f2), including Swiss legal NLP
tasks + LEXam, SecretStr credential storage, rolling-perplexity and
bits-per-byte fixes, and multilingual task loading in vLLM.

Conflicts resolved in:
- metrics/__init__.py: kept the fork's permissive batched-output handling
  (extracts all submetric keys) to support the fork's custom batched metrics.
- metrics/utils/llm_as_judge.py: kept the fork's max_tokens int fix (not the
  upstream (max_new_tokens,) tuple) and took upstream's SecretStr api_key /
  base_url plumbing.